Smooth maximum

 ( Mathematical Notation ) 

In mathematics, a smooth maximum of an indexed family x₁, ...,  x _n of numbers is a smooth approximation to the maximum function $\backslash max(x\_1,\backslash ldots,x\_n),$ meaning a parametric family of functions $m\_\backslash alpha(x\_1,\backslash ldots,x\_n)$ such that for every , the function is smooth, and the family converges to the maximum function as . The concept of smooth minimum is similarly defined. In many cases, a single family approximates both: maximum as the parameter goes to positive infinity, minimum as the parameter goes to negative infinity; in symbols, as and as . The term can also be used loosely for a specific smooth function that behaves similarly to a maximum, without necessarily being part of a parametrized family.

---

Examples

---

Boltzmann operator For large positive values of the parameter $\backslash alpha\; >\; 0$, the following formulation is a smooth, differentiable approximation of the maximum function. For negative values of the parameter that are large in absolute value, it approximates the minimum.

$$

$\backslash mathcal\{S\}\_\backslash alpha$ has the following properties:

1. $\backslash mathcal\{S\}\_\backslash alpha\backslash to\; \backslash max$ as $\backslash alpha\backslash to\backslash infty$
2. $\backslash mathcal\{S\}\_0$ is the arithmetic mean of its inputs
3. $\backslash mathcal\{S\}\_\backslash alpha\backslash to\; \backslash min$ as $\backslash alpha\backslash to\; -\backslash infty$

The gradient of $\backslash mathcal\{S\}\_\{\backslash alpha\}$ is closely related to softmax function and is given by

$$

\nabla_{x_i}\mathcal{S}_\alpha (x_1,\ldots,x_n) = \frac{e^{\alpha x_i}}{\sum_{j=1}^n e^{\alpha x_j}} 1.

This makes the softmax function useful for optimization techniques that use gradient descent.

This operator is sometimes called the Boltzmann operator, after the Boltzmann distribution.

---

LogSumExp Another smooth maximum is LogSumExp:

$\backslash mathrm\{LSE\}\_\backslash alpha(x\_1,\; \backslash ldots,\; x\_n)\; =\; \backslash frac\{1\}\{\backslash alpha\}\; \backslash log\; \backslash sum\_\{i=1\}^n\; \backslash exp\; \backslash alpha\; x\_i$

This can also be normalized if the $x\_i$ are all non-negative, yielding a function with domain $[0,\backslash infty)^n$ and range $[0,\; \backslash infty)$:

$g(x\_1,\; \backslash ldots,\; x\_n)\; =\; \backslash log\; \backslash left(\; \backslash sum\_\{i=1\}^n\; \backslash exp\; x\_i\; -\; (n-1)\; \backslash right)$

The $(n\; -\; 1)$ term corrects for the fact that $\backslash exp(0)\; =\; 1$ by canceling out all but one zero exponential, and $\backslash log\; 1\; =\; 0$ if all $x\_i$ are zero.

---

Mellowmax The mellowmax operator is defined as follows:

$\backslash mathrm\{mm\}\_\backslash alpha(x)\; =\; \backslash frac\{1\}\{\backslash alpha\}\; \backslash log\; \backslash frac\{1\}\{n\}\; \backslash sum\_\{i=1\}^n\; \backslash exp\; \backslash alpha\; x\_i$

It is a non-expansive operator. As $\backslash alpha\; \backslash to\; \backslash infty$, it acts like a maximum. As $\backslash alpha\; \backslash to\; 0$, it acts like an arithmetic mean. As $\backslash alpha\; \backslash to\; -\backslash infty$, it acts like a minimum. This operator can be viewed as a particular instantiation of the quasi-arithmetic mean. It can also be derived from information theoretical principles as a way of regularizing policies with a cost function defined by KL divergence. The operator has previously been utilized in other areas, such as power engineering.

---

Connection between LogSumExp and Mellowmax

LogSumExp and Mellowmax are the same function differing by a constant $\backslash frac\{\backslash log\; \{n\}\}\{\backslash alpha\}$. LogSumExp is always larger than the true max, differing at most from the true max by $\backslash frac\{\backslash log\; \{n\}\}\{\backslash alpha\}$ in the case where all n arguments are equal and being exactly equal to the true max when all but one argument is $-\backslash infty$. Similarly, Mellowmax is always less than the true max, differing at most from the true max by $\backslash frac\{\backslash log\; \{n\}\}\{\backslash alpha\}$ in the case where all but one argument is $-\backslash infty$ and being exactly equal to the true max when all n arguments are equal.

---

p-Norm

Another smooth maximum is the p-norm:

$$

\| (x_1, \ldots, x_n) \|_p = \left( \sum_{i=1}^n |x_i|^p \right)^\frac{1}{p}

which converges to $\backslash |\; (x\_1,\; \backslash ldots,\; x\_n)\; \backslash |\_\backslash infty\; =\; \backslash max\_\{1\backslash leq\; i\backslash leq\; n\}\; |x\_i|$ as $p\; \backslash to\; \backslash infty$.

An advantage of the p-norm is that it is a norm. As such it is scale invariant (homogeneous): $\backslash |\; (\backslash lambda\; x\_1,\; \backslash ldots,\; \backslash lambda\; x\_n)\; \backslash |\_p\; =\; |\backslash lambda|\; \backslash cdot\; \backslash |\; (x\_1,\; \backslash ldots,\; x\_n)\; \backslash |\_p$, and it satisfies the triangle inequality.

---

Smooth maximum unit

The following binary operator is called the Smooth Maximum Unit (SMU):

$$

\begin{align} \textstyle\max_\varepsilon(a, b) &= \frac{a + b + |a - b|_\varepsilon}{2} \\ &= \frac{a + b + \sqrt{(a - b)^2 + \varepsilon}}{2} \end{align} where $\backslash varepsilon\; \backslash geq\; 0$ is a parameter. As $\backslash varepsilon\; \backslash to\; 0$, $|\backslash cdot|\_\backslash varepsilon\; \backslash to\; |\backslash cdot|$ and thus $\backslash textstyle\backslash max\_\backslash varepsilon\; \backslash to\; \backslash max$.

---

See also

• LogSumExp
• Softmax function
• Generalized mean

https://www.johndcook.com/soft_maximum.pdf

M. Lange, D. Zühlke, O. Holz, and T. Villmann, "Applications of lp-norms and their smooth approximations for gradient based learning vector quantization," in Proc. ESANN, Apr. 2014, pp. 271-276. (https://www.elen.ucl.ac.be/Proceedings/esann/esannpdf/es2014-153.pdf)

Categories: Mathematical Notation, Basic Concepts In Set Theory

Post Comment